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Docket No.: 1359.1052 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re the Application of: 
Satoru WATANAB E. et al. 

Serial No. 09/940,522 Group Art Unit: 2655 

Confirmation No. 4691 

Fi.ed: August 29, 2001 hammer: WOZNIAK, JAMES S 

For. VOICE INTERACTIVE SYSTEM AND VOICE INTERACTIVE METHOD 

AMENDMENT 

Commissioner for Patents 
PO Box 1450 

Alexandria. VA 22313-1450 
Sir. 

This is in response to the Office Action mailed November 7, 2005, and having a period 
for response set to expire on February 7, 2006. 

The following amendments and remarks are respectfully submitted. Reconsideration of 
the claims is respectfully requested. 
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IN THE CLAIMS: 

The text of all pending claims, (including withdrawn claims) is set forth below. Cancelled 
and not entered claims are indicated with claim number and status only. The claims as listed 
below show added text with underlining and deleted text with otrikothrough . The status of each 
claim is indicated with one of (original), (currently amended), (cancelled), (withdrawn), (new), 
(previously presented), or (not entered). 

Please AMEND 9 and 10 in accordance with the following: 

1 . (CANCELED) 



2. (CANCELED) 

3. (PREVIOUSLY PRESENTED) A voice interactive system, comprising: 

a voice information input part inputting voice information of a first user from a first user 
terminal; 

a voice recognition part conducting voice recognition with respect to the voice information 
and analyzing contents of the voice information; 

a voice information mediation part controlling a transmission path of the voice 
information in accordance with the analyzed contents of the voice information; 

an interaction engine extracting contents of a response corresponding to the voice 
information by referring to a knowledge database and creating a synthesized voice in 
accordance with the extracted contents of the response; and 

a voice information output part outputting the synthesized voice, wherein the voice 
information mediation part monitors at all times an average reaction time, from a response of the 
interaction engine to a reaction of the first user, and in each of a case where the average 
reaction time exceeds a first threshold value that is an upper limit of a reaction time in an 
ordinary interaction and a case where the average reaction time is below a second threshold 
value that is a lower limit of the reaction time in the ordinary interaction, determines that an 
interaction between the first user and the interaction engine is not being smoothly conducted and 
allows a third-party user to participate in the interaction between the first user and the interaction 
engine, as a helper, from a terminal other than the first user terminal. 



4. (CANCELED) 
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5. (PREVIOUSLY PRESENTED) A voice interactive system, comprising: 
a voice information input part inputting voice information of a first user from a first user 
terminal; 

a voice recognition part conducting voice recognition with respect to the voice information 
and analyzing contents of the voice information; 

a voice information mediation part controlling a transmission path of the voice 
information in accordance with the analyzed contents of the voice information; 

an interaction engine extracting contents of a response corresponding to the voice 
information by referring to a knowledge database and creating a synthesized voice in 
accordance with the extracted contents of the response; and 

a voice information output part for outputting the synthesized voice, the voice information 
mediation part: 

monitoring at all times whether or not an interaction between the first user and an 
interaction engine is being smoothly conducted and, in a case of determining that the 
interaction is not being smoothly conducted, allowing a third-party user to participate in 
the interaction between the first user and the interaction engine, as a helper, from a 
terminal other than the first user terminal; 

determining a progress of interaction in accordance with an interaction time from 
a beginning of the interaction between the first user and the interaction engine and the 
number of accesses from the first user terminal to the interaction engine; and 

changing a participation mode of the third-party user successively, in an 
increasing order of progress of interaction, from (1) involvement, to (2) parallel input, and 
to (3) switching, and in a case where the participation mode is (1) involvement, permitting 
the third-party user to be involved in the interaction between the first user and the 
interaction engine, updating contents of the interaction from the interaction engine to the 
first user before the content of the interaction is output to the first user, in a case where 
the participation mode is (2) parallel input, permitting the third-part user to conduct an 
input with respect to the interaction engine in parallel with the first user, and in a case 
where the participation mode is (3) switching, permitting the third-part user to directly 
interact with the first user in place of the interaction engine. 
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6 (PREVIOUSLY PRESENTED) Avoice interactive system according to claim 3. 
wherein ihe interaction engine further includes an interaction history information storage part for 
recording interaction history on a first user basis, and a helper selection part for selecting the 
third-parry user that is considered to be most familiar with the contents of the interaction from the 
interaction history, as a helper, and the helper most appropriate for the contents of the vo.ce 
information is selected. 

7. (PREVIOUSLY PRESENTED) A voice interactive system, comprising: 
a voice information input part inputting voice information of a first user from a first user 
terminal; 

a voice recognition part conducting voice recognition with respect to the voice information 
and analyzing contents of the voice information; 

a voice information mediation part controlling a transmission path of the voice 
information in accordance with the analyzed contents of the voice information; 

an interaction engine extracting contents of a response corresponding to the voice 
information by referring to a knowledge database and creating a synthesized voice in 
accordance with the extracted contents of the response; and 

a voice information output part outputting the synthesized voice, wherein the voice 
information mediation part monitors at all times whether or not an interaction between the first 
user and the interaction engine is being smoothly conducted and, in a case of determining that 
the user's interaction is not being smoothly conducted, allows a third-party user to participate in 
the interaction between the user and the interaction engine, as a helper, from a terminal other 

than the first user terminal; 

a help request notification part operative, in a case where the voice information 
mediation part determines that the interaction is not being smoothly conducted, to notify a third- 
party helper user of such fact, and: 

in a case where the help request notification part notifies the third-party user of the fact 
that the first user's interaction is not being smoothly conducted, the third-party user is capable of 
voluntarily interacting with the first user, and 

in a case where it is detected that only a voice of the third-party helper user continues, for 
a predetermined period of time or longer, in an interaction between the third-party user and the 
first user, the interaction engine interacts only with the third-party user. 
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8. (PREVIOUSLY PRESENTED) A voice interactive system according to claim 3, 
further comprising 

an interaction histoiy display part displaying the interaction history stored in the 
interaction history information storage part to a third-party user; and 

a helper instruction part for receiving a help instruction from the third-party helper user, 

wherein: 

when the help instruction part receives the help instruction from the third-party 
helper user, the voice information mediation part enables the interaction between the 
third-party helper user and the user to be conducted, and 

when a degree of help of the third-party helper user exceeds a predetermined 
threshold value in interaction between the third-party helper user and the first user, the 
interaction engine interacts only with the third-party helper user. 

9. (CURRENTLY AMENDED) A voice interactive method, comprising: 
inputting a first user's voice information from a first user terminal; 
conducting voice recognition with respect to the voice information, and analyzing 
contents of the voice information; 

controlling a transmission path of the voice information in accordance with the analyzed 

contents of the voice information; 

outputting a synthesized voice; 

controlling a transmission path of the voice information, comprising: 

extracting contents of a response corresponding to the voice information by referring to 

a knowledge database, and creating a synthesized voice in accordance with the contents of the 

response, and 

m onitoring ot oil time"- whether or not th n Hoor'c I nteraction io boing cm oo th iy 
e u n d u a odand, in o caco whoro it is d r tom l nod thot the user's i nteracti o n is not bein g ^m on th l y 
uj i id uotod, allow i n g n third party uccr t o p n rt i cipato, ac i h nl pu, in the inf mrrti on b o tweon th e 
foot user and an interaction onginc from anothor terminal .monitoring at all times an avera ge 
rairtmn time, from a response of the interaction en gi ne to a rea ction Q f the first user, apa" jn 
**rh nf * case where ft* average reac t ?™ limn speeds a first threshold value that is an upper 
limit nf a reaction tim* in an ordinary i nteraction and a case where the average reaction time is 
hPlnw a second threshold value that is a lower li mit of the rea ction time in the ordina r y 
intamntinn. determine that an inter a ction between the first user and the interaction engine is 
not being smoothly conducted and allowing a third-party user to participate in the interaction , 
between the first user and the interaction e n nine. as a helper, from a terminal other than the first 
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user terminal. 

10. (CURRENTLY AMENDED) A computer-readable medium storing a program to 
be read and executed by a computer for processing an input user's voice Information, by. 
inputting a first user's voice information; 

conducting voice recognition with respect to the voice information, and analyzing 

contents of the voice information; 

controlling a transmission path of the voice information in accordance with the analyzed 

contents of the voice information; and 

outputting a synthesized voice, wherein: 

in the controlling of a transmission path of the voice information, contents of a 
response corresponding to the voice information being extracted by referring to a 
knowledge database, and a synthesized voice being created in accordance with the 
contents of the response, and 

in the controlling of the transmission path of the voice information, monitoring at 
al l ttmoc whcthor or not tho user's interaction is being omoothly conducted and, in o n ww 
whoro it io dotormin orl thattho uoor'o interaction i c not boing smoothly conducted, 
jl luwi ng 3 t hir d pnrty i nr rtn rnrt-'r-*" i " - h "'P™- in ih0 interaction botwocn tho firct 
uhcr and an interaction eng i ne from another torm i nol.monitoring at all times an average 
P ff ^Hnn time, from a resnonse of the inte r action engine to a reaction of the first user, an d 
jn aa rh rrf a rase where the average r e union time exceeds a first threshold value that is 
an np ppr limit of a reaction time in an o rdinary interaction and a case where the average 
r^rtinn time is below a second threshold value th at is a lower limit of the reaction time in 
the ordinary interaction, determining that an interaction between the first user and the 
interaction engine is not being smoothly con d uced and allowing a third-party user to 
partici pate in the interaction between the firs t user and the interaction engine, as a 
helper, from a terminal other than the first use r terminal. 

1 1 (PREVIOUSLY PRESENTED) A voice interactive system according to claim 5, 
wherein the interaction engine further includes an interaction history information storage part for 
recording interaction history on a first user basis, and a helper selection part for selecting the 
third-party user that is considered to be most familiar with the contents of the interaction from the 
interaction history, and most appropriate for the contents of the voice information. 
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12. (PREVIOUSLY PRESENTED) A voice interactive system according to claim 7, 
wherein the interaction engine further includes an interaction history information storage part for 
recording interaction history on a first user basis, and a helper selection part for selecting the 
third-party user that is considered to be most familiar with the contents of the interaction from the 
interaction history, and most appropriate for the contents of the voice information. 

1 3. (PREVIOUSLY PRESENTED) A voice interactive system according to claim 5, 
further comprising: 

an interaction history display part displaying the interaction history stored in the 
interaction history information storage part to a third-party user; and 

a helper instruction part for receiving a help instruction from the third-party helper user, 
wherein: 

when the help instruction part receives the help instruction from the third-party 
helper user, the voice information mediation part enables the interaction between the 
third-party helper user and the user to be conducted, and 

when a degree of help of the third-party helper user exceeds a predetermined 
threshold value in interaction between the third-party helper user and the first user, the 
interaction engine interacts only with the third-party helper user. 

14. (ORIGINAL) A voice interactive system according to claim 7, further comprising 
an interaction history display part displaying the interaction history stored in the interaction 
history information storage part to a third-party user; and 

a helper instruction part for receiving a help instruction from the third-party helper user, 
wherein: 

when the help instruction part receives the help instruction from the third-party 
helper user, the voice information mediation part enables the interaction between the 
third-party helper user and the user to be conducted, and 

when a degree of help of the third-party helper user exceeds a predetermined 
threshold value in interaction between the third-party helper user and the first user, the 
interaction engine interacts only with the third-party helper user. 
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15. (CURRENTLY AMENDED) A voice interactive method, comprising: 

inputting voice information of a first user from a first user terminal; 

conducting voice recognition with respect to the voice information and analyzing contents 

of the voice information; 

controlling a transmission path of the voice information in accordance with the analyzed 

contents of the voice information; 

extracting contents of a response corresponding to the voice information by referring to a 

knowledge database and creating a synthesized voice in accordance with the extracted contents 

of the response; and 

outputting the synthesized voice, the voice information mediation part: 

monitoring at all times whether or not an interaction between the first user and an 
interaction engine is being smoothly conducted and, in a case of determining that the 
interaction is not being smoothly conducted, allowing a third-party user to participate in 
the interaction between the first user and the interaction engine, as a helper, from a 
terminal other than the first user terminal : 

determining a progress of interaction in accordance with an interaction time from 
a beginning of the interaction between the first user and the interaction engine and the 
number of accesses from the first user terminal to the interaction engine; and 

changing a participation mode of the third-party user successively, in an 
increasing order of progress of interaction, from {1} involvement, to (2) parallel input, and 
to (3) switching, and in a case where the participation mode is (1 ) involvement permitting 
the third-party user to be involved in the interaction between the first user and the 
interaction engine, updating contents of the interaction from the interaction engine to the 
first user before the content of the interaction is output to the first user, in a case where 
the participation mode is (2) parallel input, permitting the third-part user to conduct an 
input with respect to the interaction engine in parallel with the first user, and in a case 
where the participation mode is (3) switching, permitting the third-part user to directly 
interact with the first user in place of the interaction engine. 



8 
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16. (CURRENTLY AMENDED) A computer-readable medium storing a program to 

be read and executed by a computer for processing an input user's voice information, by: 
inputting voice information of a first user from a first user terminal; 
conducting voice recognition with respect to the voice information and analyzing contents 

of the voice information; 

controlling a transmission path of the voice information in accordance with the analyzed 

contents of the voice information; 

extracting contents of a response corresponding to the voice information by referring to a 

knowledge database and creating a synthesized voice in accordance with the extracted contents 

of the response; and 

outputting the synthesized voice, the voice information mediation part; 

monitoring at all times whether or not an interaction between the first user and an 
interaction engine is being smoothly conducted and, in a case of determining that the 
interaction is not being smoothly conducted, allowing a third-party user to participate in 
the interaction between the first user and the interaction engine, as a helper, from a 
terminal other than the first user terminal ; 

determining a progress of interaction in accordance with an interaction time from 
a beginning of the interaction between the first user and the interaction engine and the 
number of accesses from the first user terminal to the interaction engine; and 

changing a participation mode of the third-party user successively, in an 
increasing order of progress of interaction, from (1 ) involvement, to (2) parallel input, and 
to (3) switching, and in a case where the participation mode is (1) involvement, permitting 
the third-party user to be involved in the interaction between the first user and the 
interaction engine, updating contents of the interaction from the interaction engine to the 
first user before the content of the interaction is output to the first user, in a case where 
the participation mode is (2) parallel input, permitting the third-part user to conduct an 
input with respect to the interaction engine in parallel with the first user, and in a case 
where the participation mode is (3) switching, permitting the third-part user to directly 
interact with the first user in place of the interaction engine. 
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17. (PREVIOUSLY PRESENTED) A voice interactive method, comprising: 

inputting voice information of a first user from a first user terminal; 

conducting voice recognition with respect to the voice information and analyzing contents 

of the voice information; 

controlling a transmission path of the voice information in accordance with the analyzed 

contents of the voice information; 

extracting contents of a response corresponding to the voice information by referring to a 
knowledge database and creating a synthesized voice in accordance with the extracted contents 

of the response; and 

outputting the synthesized voice, while monitoring at all times whether or not an 
interaction between the first user and the interaction engine is being smoothly conducted and, in 
a case of determining that the user's interaction is not being smoothly conducted, allowing a 
third-party user to participate in the interaction between the user and the interaction engine, as a 
helper, from a terminal other than the first user terminal, and: 

in a case where the interaction is not being smoothly conducted, notifying a third- 
party helper user of such fact, 

in a case where the third-party user is notified of the feet that the first user's 
interaction is not being smoothly conducted, the third-party user is capable of voluntarily 
interacting with the first user, and 

in a case where it is detected that only a voice of the third-party helper user 
continues, for a predetermined period of time or longer, in an interaction between the 
third-party user and the first user, controlling the interaction engine to interact only with 
the third-party user. 

1 8. (PREVIOUSLY PRESENTED) A computer-readable medium storing a program 
to be read and executed by a computer for processing an input user's voice information, by: 
inputting voice information of a first user from a first user terminal; 
conducting voice recognition with respect to the voice information and analyzing contents 

of the voice information; 

controlling a transmission path of the voice information in accordance with the analyzed 

contents of the voice information; 

extracting contents of a response corresponding to the voice information by referring to a 
knowledge database and creating a synthesized voice in accordance with the extracted contents 
of the response; and 

outputting the synthesized voice, while monitoring at all times whether or not an 
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interaction between the first user and the interaction engine Is being smoothly conducted and, in 
a case of determining that the user's interaction is not being smoothly conducted, altowmg a 
third-party user to participate in the Interaction between the user and the interaction engme, as a 
helper from a terminal other than the first user terminal, and: 

in a case where the interaction is not being smoothly conducted, notifying a th.rd- 

party helper user of such fact, 

in a case where the third-party user is notified of the fact that the first user's 
interaction is not being smoothly conducted, the third-party user is capable of voluntarily 
interacting with the first user, and 

in a case where it is detected that only a voice of the third-party helper user continues, for 
a predetermined period of time or longer, in an interaction between the third^arty user and the 
first user, controlling the interaction engine to interact only with the third-party user. 
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REMARKS 

In accordance with the foregoing, claims 1,2, and 4 were previously cancelled, claims 3, 
5, 7, and 9-16 are currently amended, as detailed hereinafter, to clarify the distinction over the 
prior art and rejections of the Action. No new matter is presented and, accordingly, approval and 
entry of the amended claims are respectfully requested. 

STATUS OF CLAIMS 

Claims 3 and 5-18 are pending the application. Claims 7, 12, 14, 17 and 18 are allowed. 
Claims 3, 5,6, 8-11, 13, 15 and 16 are rejected. 

ITEM 2: EXAMINER'S RESPONSE REGARDING ARGUMENTS AS TO CLAIMS 5 AND 15-16 

Applicants argued that as to claims 5, 15 and 16, the prior art of record fails To teach a 
progressive operator interaction (Amendment, pages 15-16). In response, the Examiner urges 
that Davis et al. 'teaches such a progressive interaction in the form of a standard update 
(involvement) of the user entry form at an operator station (col. 9, lines 1-11), ...[and]... direct 
interaction (parallel input of data) through data (col. 11, lines 36-58)... It is respectfully 
submitted that the Examiner misinterprets the terminology of claim 5. 

In claim 5, "involvement" means a state where "the third party (operator) can update the 
contents of interaction between the first user and the interaction engine", and the "parallel input" 
means the state where Ihe third party can conduct an input to the interaction engine in parallel 
with the first user. The meanings of "involvement" and "parallel input" which should be given to 
the terms of the claims are not disclosed in col. 1 1 , line 36, col. 1 2, line 20 of Davis and, 
accordingly, Applicants' prior arguments are submitted to be adequate, alone, to distinguish over 
the prior rejection. 

ITEM 5: REJECTION OF CLAIMS 5, 11, 13 AND 15-16 FOR OBVIOUSNESS OVER 
BOHACEK ET AL. IN VIEW OF MARX ET AL. (U.S. PATENT 6,173,266) AND DAVIS ET AL. 
(U.S. PATENT 5,583,922) 

The rejection is respectfully traversed. 

In mounting this new ground of rejection, the Examiner does not compare the limitation of 
claim 5 "determines a progress of interaction in accordance with an interaction time from a 
beginning of the user's interaction and the number of accesses to the interaction engine" with 
the cited Marx reference in a correct manner. To the contrary, the Examiner erroneously 
describes the limitation of claim 3 at page 8, line 7 from the bottom to the last line on page 8 of 
the Office Action. The rejection is fatally defective, since not addressing terminology of claim 5. 
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More particularly, as here pertinent, the Examiner points out that Daudelin discloses in 

col. 11, lines 23-54 that: 

..in a case where the average reaction time exceeds a first threshold value 
that is an upper limit of a reaction time in ord inary interaction or in a case 
where the average reaction time is below a second threshold value that is a 
lower limit of a reaction time in ord inary interaction, allows a third-party user 
to participate". 

(Action at page 5, third paragraph; emphasized recitations appear in Daudelin but 
not in the Examiner's citation of same) 

To the contrary of the Examiner's interpretation of Daudelin, quoted above, Daudelin 
describes in col. 1 1 , lines 26-28 that "if no response is made by the called customer within a 
predetermined interval". This description may correspond to the clause of the claims of "in a 
case where the average reaction time exceeds a first threshold value that is an upper limit of a 
reaction time in ordinary interaction". However, Daudelin does not disclose the further recitation, 
common to claims 3, 9 and 1 0, that "in a case where the average reaction time is below a 
second threshold value that is a lower limit of a reaction time in ordinary interaction". 

As recited in claims 3, 9 and 1 0 f by allowing a third party user to participate "in a case 
where the average reaction time is below a second threshold value that is a lower limit of a 
reaction time in ordinary interaction", an operator (third party) is allowed to appropriately 
participate in accordance with an interaction situation of a user, using the nature peculiar to 
interaction: "when interaction becomes redundant, a reaction time is likely to become short 
gradually (page 10, lines 7-18 of the specification)". 

Accordingly, independent claims 3, 9 and 10 are submitted to distinguish patentably over 
the references relied upon in Item 4 of the Action, as applied to claims 3, 6 and 8 as well as to 
claims 9 and 10 as hereinabove explained, whether those references are taken singularly or in 
any proper combination. 

LACK OF PRIMA FACIE DEMONSTRATION OF OBVIOUSNESS OF THE COMBINATION OF 
PRIOR ART REFERENCES RELIED UPON RENDERS ALL OF THE REJECTIONS FATALLY 
DEFECIENT 

In each of the foregoing combinations, the basic thrust of the justification for the 
combination resides in the contention that "it would have been obvious to a person of ordinary 
skill in the art..." to modify the teachings of one reference with those of another - - and r indeed, 
Item 4 relies on Bohacek, Daudelin and Davis whereas Item 5 relies on the different combination 
of Bohacek, Marx and Davis, the latter being urged to be an obvious combination on grounds 
equally deficient as those advanced in Item 4 of the Action. MPEP 2143.2143.03. 
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reaction time interval, however, Daudelin recites.... 
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